Table of Contents [Sticky]

Video Stabilization process using Optical Flow

Optical Flow Fundamentals

Optical flow is a computer vision technique used to estimate the apparent motion of image features between consecutive frames of a video sequence. By analyzing how pixels move from one frame to the next, optical flow provides information about the relative motion of objects and the camera.

The fundamental assumption behind optical flow is the brightness constancy constraint, which states that the intensity of a point remains approximately constant as it moves between frames. If a pixel located at $(x, y)$ at time $t$ moves by $(u, v)$ during a time interval $Δ t$ , then:

$I (x, y, t) = I (x + u, y + v, t + Δ t)$

Assuming small displacements and applying a first-order Taylor approximation leads to the optical flow constraint equation:

$I_{x} u + I_{y} v + I_{t} = 0$

where:

$I_{x}$ and $I_{y}$ are the image intensity gradients.
$I_{t}$ is the temporal intensity variation.
$u$ and $v$ represent the horizontal and vertical motion components.

Algorithms such as Lucas-Kanade use this constraint to track feature points across consecutive frames and estimate their motion vectors.

Optical Flow for Video Stabilization

In Digital Image Stabilization (DIS), optical flow is used to estimate camera motion directly from the video stream. First, a set of feature points is detected and tracked between consecutive frames using an optical flow algorithm. This produces a collection of point correspondences that describe how image features move over time.

These local motion estimates are then used to compute a global motion model. A common approach is to estimate a homography, represented by a 3×3 projective transformation matrix:

$H = [\begin{matrix} h_{11} & h_{12} & h_{13} \\ h_{21} & h_{22} & h_{23} \\ h_{31} & h_{32} & h_{33} \end{matrix}]$

The homography describes how points in one frame map to points in the next frame and can model translation, rotation, scaling, shearing, and perspective effects. Given a point $𝐩$ in homogeneous coordinates, its transformed position is obtained as:

$𝐩^{'} = H 𝐩$

By estimating a homography for each pair of consecutive frames, the stabilization algorithm recovers the camera trajectory throughout the video. This trajectory is then smoothed to remove high-frequency shake while preserving intentional motion. Finally, the inverse correction is applied to each frame through image warping, producing a stabilized video sequence.

Stabilization Pipeline

Video frames
    |
    v
Optical Flow Tracking
    |
    v
Homography Estimation
    |
    v
Smooth motion
    |
    v
Image warping
    |
    v
Video frame with motion correction

❯

Share This Page

Send Email Using

Video Stabilization process using Optical Flow

Optical Flow Fundamentals

Optical Flow for Video Stabilization

Stabilization Pipeline